Security

None.

Review metrics

Merge-risk options

Maintainer options:

1. Preserve adapter-aware state scoring (recommended)
   Pass canonical state-query results from the live adapter into completion scoring and cover a stateful Hermes task before merge.
2. Narrow local adapter support
   If stateful Hermes tasks are intentionally out of scope, explicitly filter them and document the files-and-execution-only support boundary.

@clawsweeper automerge

Special instructions:
Route live adapter state-query results into completion scoring before teardown, preserving capability-missing handling and adding a stateful Hermes regression test.

Technical review

Best possible solution:

Carry active-adapter state-query results into completion scoring before teardown, preserving the neutral capability-missing policy and adding a stateful Hermes regression test.

Do we have a high-confidence way to reproduce the issue?

Yes. A Hermes AIAgent-mode task requiring memory or cron reaches the PR’s local adapter path, which admits the capability but scores it using _LocalVerificationClient rather than adapter state verification.

Is this the best way to solve the issue?

No. The local lifecycle is the right direction, but it must feed the adapter’s canonical state-query outcomes into scoring instead of treating them as Gateway RPC checks.

Full review comments:

• [P1] Route state assertions through the active adapter — clawbench/harness.py:640-651
_run_single_adapter tears down the adapter and then calls score_task_run with _LocalVerificationClient, without ever invoking adapter.verify_state_query. Hermes AIAgent mode admits memory and cron tasks, so their state assertions are evaluated as unavailable Gateway RPCs instead of Hermes state. This is the unchanged prior P1 blocker.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded benchmark-validity defect affecting local adapter runs rather than the core OpenClaw execution path.
• merge-risk: 🚨 other: Merging would permit benchmark results whose supported adapter-state assertions are scored through the wrong verification boundary.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body records an after-fix local Hermes run improving the named task from 0.63704 to 1.000; the remaining blocker is incorrect stateful scoring, not missing proof.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body records an after-fix local Hermes run improving the named task from 0.63704 to 1.000; the remaining blocker is incorrect stateful scoring, not missing proof.

Evidence

Acceptance criteria:

• [P1] pytest tests/test_harness.py tests/test_hermes_adapter.py -q.

What I checked:

• Current main has not superseded the capability: Current main explicitly rejects registered adapters that are not OpenClaw for executable harness runs, so Hermes local execution remains unimplemented on main. (clawbench/harness.py:125, 884dd1bb5511)
• PR scores through the wrong verification boundary: The PR creates and runs the adapter but tears it down before calling score_task_run with _LocalVerificationClient; it never invokes adapter.verify_state_query for canonical state queries. (clawbench/harness.py:640, 67ff264005ab)
• Canonical contract requires adapter state verification: The canonical verifier contract says state queries are resolved by adapter.verify_state_query, separately from workspace file and execution checks. (clawbench/canonical/schema.py:165, 884dd1bb5511)
• Hermes has supported stateful paths: In AIAgent mode Hermes advertises memory and cron capabilities and implements corresponding state-query handling, which the PR bypasses during scoring. (clawbench/adapters/hermes.py:249, 884dd1bb5511)
• Prior blocker remains at the same PR head: The current PR head is the same commit previously reviewed; its parent diff shows the state-scoring path originates in the first PR commit, while the current head only adds Hermes prompt guidance. (clawbench/harness.py:640, 67ff264005ab)
• Adapter-contract provenance: The current Hermes state-query implementation was introduced with adapter canonicalization. (clawbench/adapters/hermes.py:598, 56531fbf432b)

Likely related people:

• scoootscooob: Introduced the canonical adapter and Hermes state-query behavior that this harness path needs to use. (role: introduced adapter verification contract; confidence: high; commits: 56531fbf432b; files: clawbench/adapters/hermes.py, clawbench/adapters/base.py, clawbench/canonical/schema.py)
• Vincent Koc: Recent current-main commits cover harness runtime and scoring behavior adjacent to this integration. (role: recent harness and scoring contributor; confidence: medium; commits: ed9adf8d8451, fb486a1ed307; files: clawbench/harness.py, clawbench/scorer.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Route live adapter state-query results into scoring before teardown.
• Add a stateful Hermes regression test covering an admitted capability.

Rating scale

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History